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ABSTRACT 

In 1981, the Pittsburgh (Pennsylvania) Board of 
Education contracted with an External Review Team composed of 
educators of national stature representing diverse perspectives and 
experiences to realize a portion of its mission statement, "To 
provide the board of education, the superintendent of schools, and 
the general public of Pittsburgh with an objective assessment of: (1) 
the priorities set by the board of education since 1980; (2) the 
programmatic responses of the superintendent and administration to 
those priorities; (3) the implementation and evolution of these 
programmatic responses and their integration into the educational 
program by school district staff; and (4) the status of the overall 
educational program compared to other public school districts." The 
educational program audit examines a large number of individual 
program-focused activities. As an evaluation tool of educational 
programs, it reviews common themes, outcomes, and problems. This 
report describes the motivation behind the Pittsburgh School 
District's educational program audit, the manner in which it was 
designed and conducted, and the benefits realized from the effort. In 
addition, the External Review Team members share their perceptions of 
the audit, analyze its pitfalls, state how to avoid pitfalls, and 
recommend how to realize audit objectives. The review team found 
that: (1) Pittsburgh's children are well served by their public 
schools; (2) outstanding professionals stiff the city's schools; and 
(3) school programs are relevant because of systematic and 
coordinated planning, student achievement score gains reflect the 
successes of educational programs in the Pittsburgh school district. 
(JAM) 
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Seeing the Whole Picture: The Educational Program Audit 



Introduction 

In spring 1988, the Pittsburgn Board of Public Education commissioned what it 
called an "educational program audit. " This audit was conceived as a unique 
form of evaluation, one that would permit a wide-ranging review of all of the 
district's efforts pertaining to educational programming. The scope of the 
effort inciuded curriculum, instruction, discipline, student assessment, 
educational leadership, and professionalism. What emerged was a type of 
evaluation that offered opportunities unlike any other. It permitted the 
synthesis of knowledge gained across a wide variety of program areas. It 
allowed observations about how beat to coordinate such varied programs into a 
coherent whole. 

To generalize, the educationai program audit examines a large number of 
individual program-focused activities. It notes common outcomes, themes, and 
problems that might suggest issues that are more structural than those that 
would be realized in more typical forms of evaluation. The educational program 
audit also provides commentary on how well or poorly various district efforts 
are interacting with each other to form a complete educational program. These 
evaluation outcomes represent achievements well beyond what is possible through 
the analysis of monitoring data or the program-focused evaluation that defines 
the typical work scope of departments of research and evaluation within school 
districts. For this reason, the educational program audit represents a unique 
and useful evaluation tool. To be most effective, the audit must be broadly 
focussed and must encourage and respect the professional judgment of the review 
team members as they explore issues that they determine to be important and 
offer advice to the district. 

This paper examines one such effort, describing the motivation behind it, the 
manner in which it was designed and conducted, and the benefits that were 
realized from the effort. Perhaps most valuable of all, individuals who were 
most closely involved in the exercise will share their perceptions of the 
educational program audit: its pitfalls and how they might be avoided; its 
promises and how they might be realized. 

Setting 

In 1981, the Pittsburgh Board of Education established ten priorities for the 
district. They were organized under two broad headings: School Improvement 
and Cost Effective Management. In 1986, the board reviewed and revised its 
priorities, setting them if. Tour major areas: Achievement, Fiscal 
Resoonsibil ity , Student Discipline, and Sustaining and Improving [Various] 
District Initiatives. 

In March, 1988, to get an independent perspective on the district's programs, 
the Pittsburgh Board and its superintendent contracted with an External Review 
Team to accomplish the following mission: 

Mission Statement: To provide the board of education, the 
superintendent of schools, and the general public of Pittsburgh with 
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an objective assessment of: i) tne priorities set by the board of 
education since 1980, 2) tne programmatic responses of the 
superintendent and administration to those priorities, 3) the 
implementation and evolution of tnese programmatic responses and 
their integration into the educational program of the Pittsburgh 
Public Schools by district staff, and 4) the status of the overall 
educational program compared to other public school districts. 

The mission of the Externa; heview Team aefined a special type of evaluation, 
the educational program audit. An auoit occurs to verify claims made by a 
practitioner. In this case, the auait focused on the educational program that 
had been developea by the superintendent and administration of the Pittsburgh 
Public Schools since 1980 in response to board priorities. The work of the 
review team was not, however, limited to simple review of district practice in 
an attempt to establish a correspondence between priorities, plans, and 
activities. Instead, a strong element of judgment was required, beginning with 
the selection of tne review team members. 

Those selected to serve on the Externa^ Review Team were educators with 
national stature, representing diverse perspectives and experiences. Their 
charge was not the review of a single project or collection of projects or of a 
single report or series of reports. Rather, it called for a critical 
inspection of the district's entire educational program, particularly as it 
relateo to the board's priorities. This meant that the review team was invited 
to comment on the spirit of the board priorities as well as their literal 
content. The element of judgment was valued and "texture" or "feel" was a 
legitimate focus, just as data and written reports were. In short, this audit 
was district-initiated, comprehensive, and focused on the relatively long-term 
development of an educational program. 

Assembling the External Review Team 

The External Review Team was selectee in two stages. First, the superintendent 
and the district's Director of Researcn, Evaluation, and Test Development (who 
served as the district's liaison to tne review team) chose the chair of the 
review team, Peirce Hammond, Deputy Executive Director of the Southeastern 
Educational Improvement Laboratory. Hammond, an educator known to both of 
them, had urban education experience in two large districts, including service 
as the head of the research and evaluation unit in Cleveland. Next, those 
three, in consultation with the President of the Pittsburgh Board of Education 
selected the balance of the team. 

The review team was constituted to represent major constituent groups. It 
included teacher, superintendent, board member, administrator, and researcher 
representation, and was also representative by race and gender. The teaching 
representative was nominated by the district liaison. He was Miles Myers, 
President of the California Federation of Teachers (and Administrative Director 
of the Bay Area, California, and National Writing Projects. The superintendent 
representative was Floretta Dukes McKenzie who retired as superintendent of the 
Washington, DC, Public Schools shortly before the audit was begun. McKenzie 
was known to the Pittsburgh superintendent, the board president, the review 
team chair and the district liaison. The board representative was Rita 



Walters, President of the Los Angeles Unified Board of Education. She was 
known to the board president and to the superintendent. To complete the review 
team, a researcner with national stature was sought. Beatrice Ward, previously 
Deputy Director of the Far West Laboratory for Educational Research and 
Development, working as an independent consultant and well known for her work 
in the area of research on teaching, was selected. She was known to the 
superintendent, the district liaison, and the review team chair. 

While the review team was selected to provide substantive balance and varied 
perspectives, the quality of its membership was such that the whole was greater 
than the sum of its parts. First, each person represented far more than the 
narrow interests of her/his group. Second, eacn was experienced and understood 
the neea and value of diverse viewpoints in tnis process. Third, each knew a 
considerable amount about educational practice in other districts to contribute 
ideas about alternative approaches and about the pitfalls and promise of the 
strategies being employed or considered in Pittsburgh. 

The Review Process 

The educational program audit mirrorea the program planning and development 
cycle of the Pittsburgh Public Schools (Wallace, 1986; LeMahieu, 1984). That 
cycle begins with a far reaching assessment of needs. This includes a 
comprehensive review and analysis of existing data as weiJ as the assessment of 
the perceptions of all major constituents regarding the conditions in the 
district most in need of attention or improvement. The results of the needs 
assessment are presented for review by the school boarc^ . typically at a retreat 
convened especially for that purpose. The board considers all relevant data 
and then establishes a set of priorities that serve as the guiding focus of the 
district's efforts for the ensuing years. Subsequently, the superintendent 
convenes committees to prepare plans to address the board's priorities, and 
program design is begun. From that point forward, all of the district's 
programming efforts and all actions submitted to the board for approval must be 
justified in terms of the board's priorities. Historically, the cycle lasts 
approximately six or seven years, before beginning anew. 

The Educational Program Audit began with an analytic review of the 
comprehensive plans developed by the administration in response to board 
priorities of 1931 and 1986 (Hammond, 1988a and 1988b). The review team agreed 
• r — with the board and superintendent to focus on four areas: Achievement, 

Discipline, Staff Evaluation, and Attracting and Holding Students. The basic 
questions to be addressed included whether the desired programs were being 
implemented in a way that was consistent with the board's priorities; whether 
they were likely to achieve the mandates established oy the board f s priorities; 
and whether they represented "state-of-the-art" practice. The next activity, 
undertaken in Spring 1988, saw the review team interview over 30 key staff 
members involved in the planning and implementation of programs designed to 
fulfill the board's priorities (Hammond, McKenzie, Myers, Walters, and Ward, 
1988). Finally, in Fall 1988, the review team observed these programs in 10 
schools (out of 72) and discussed them with over 100 district employees charged 
witL their implementation (Hammond, McKenzie, Myers, Walters, and Ward, 1989). 
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Important c haracteristics of the Educational Program Audit 



There were several characteristics that made this Educational Program Audit 
unique with respect to trie evaluation activities typically pursued in school 
districts. These characteristics contributed to the success of this 
undertaking, and make it wortn considering by others. As noted above, the 
characteristics of the audit were the fact that It was district-initiated, it 
was comprehensive in nature, and it was designed to permit a longitudinal view. 
The implication of each of tnese characteristics is discussed briefly below. 

District Initiated - Whiie scnool districts commonly undertake evaluations of 
individual projects, often this is still in response to mandates from project 
funders such as the Federal Government (e.g., Chapter 1). These evaluations 
are what King and Pechman (1984) call "signalling ... .The use of evaluation 
information as signals from the local school district to funding and 
legislative support agencies that aJl is well. 11 In the twenty-four years since 
the passage of the F.lementary and Secondary Education Act, there has been 
increasing understanding of the substantive and political value of evaluation. 
Modern superintendents make use of evaluation as a fact of life and a potential 
asset. When they do, the evaluation is "charged. ... [that is,] actively used by 
local managers...." (King and Pechman, 1984). While not all superintendents 
are sophisticated in their understanding of evaluation, there are some who are 
quite knowledgeaDle. Pittsburgh Superintendent Richard Wallace has worked hard 
to establish a climate that respects the use of evaluation and its results in 
program planning and implementation. 

Others (e.g., Cooley and Bickel, 1986; Alkin, 19??; and Cronbach, 1982) have 
observed the power of a sincere interest in the client's needs and perspectives 
in promoting the usefulness and use of evaluation. This so-called "client 
orientation" is essential not only to ensure the client's cooperation in and 
commitment to the work, but even more to assure that the evaluation 
appropriately matches district needs and is highly likely to have an effect on 
district educational practice. 

When Wallace found that a simplistic use of statistics misrepresented the 
progress he believed the district co have made between 1980 and 1988, he 
determined to ask an independent review team to review the district's efforts 
during that period. While Wallace's agenda for the audit was, consequently, 
partially defensive, [partially in response to this situation,] he was careful 
to ask the review team to be broad-ranging. He charged it to determine the 
extent to which the Pittsburgh school district's reputation for innovation and 
progress was warranted. Further, he ensured the ownership of the study by his 
school board by having members of the review team meet with the board to 
discuss plans and amend the proposed activities based on that discussion. In 
particular, the board president was included in a variety of ways that shaped 
the audit agenda: he helped select the review team; he, too, asked the team to 
shed light on the district's newly won reputation; and he raised issues and 
suggested areas of inquiry for the consideration of the review team (e.g., that 
the district's possible overemphasis of the cognitive aspects of education and 
consequent under emphasis of the social, cultural, and emotional components of 
learning) . 



Comprehensive - The comprehensiveness of this educational program audit 
distinguishes it from most program-specific evaluations in that the review team 
set out to determine the consistency of outcomes across programmatic efforts 
with common goals, discover fundamental themes across projects, and analyze 
recurrent problems in implementation. In this audit, the review team found 
that the district had, indeed, established a successful basic skills program 
that had raised student achievement and brought the district together 
psychologically. What the district faced, however, were the limitations of its 
basic skills program and the need to integrate higher cognition as well as 
social, cultural, and emotional components into its educational program. 
Furthermore, thLS need existed not simply to continue achievement gains, but to 
realize other* highly held goals, namely, placing greater emphasis on 
eliminating the racial achievement gap confronting the district and building 
the discipline program tnat had been neglected while the basic skills emphasis 
dominated. 

A different, though no less pervasive, theme emerged as the review team 
examined the district's staff evaluation program. The district's Pittsburgh 
Research-based Instructional Supervisory Model (PRISM) program, includes the 
well-known Schenley High School Teacher Center. The development of this center 
occurred largely because of the mutual respect and cooperation that developed 
among the board of education, the superintendent, and the head of the 
Pittsburgh Federation of Teachers. The PRISM program was developed because the 
superintendent argued that the board's desire to rid the district of inadequate 
teachers and administrators could only be carried out once it was established 
that: (1) each teacher and administrator had been informed as to the 
district's definition of effective instruction and effective instructional 
leadership; (2) each teacher and principal had been provided adequate 
instruction about how to be effective within the terms of that definition; and 
(3) each had been given a fresh opportunity to teach or lead effectively. 

This argument led to the establishment of a wide range of staff development 
programs (Wallace, LeMahieu, and Bickel, in press). The programs developed for 
and with teachers have been highly acclaimed and successful in the view of the 
review team as well as those of other external evaluators. Those for and with 
principals have not enjoyed similar success. Indeed, the review team found a 
need for z support system for principals, especially newly-appointed ones. 
Unfortunately, the review team found that a negative tone characterized the 
board's relationship with the district's administrators, an air of mistrust and 
uncertainty that is inconsistent with the constructive programs and support 
that have been the foundation of the district's accomplishments. 

While there is an opportunity, in a comprehensive study, to identify broad 
themes and insights that cut deeper than is possible through traditional 
project evaluations, there is still a r'sk that what is observed is not really 
"there." The risk derives from several factors: First, tnere is an ever 
present danger for all evaluators. Co-optation — being unduly influenced by 
personality or circumstance is a risk with this form of evaluation. The 
magnitude of the task taken on by those responsible for education in any major 
school district is such that sympathy comes easily. As Cooley and Bickel 
(1986) note, this kind of research, being what they call decision-oriented 
educational research, has "...the deliberate objective of generating 
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information that is of immediate use in some policy or decision context, [p. 
35]." Such researcn must walk the line oetween what decision makers want to 
hear and what they need to hear even more carefully than other kinds. Second, 
the review team will have some biases Duilt in by virtue of its composition. 
Some will occur because human beings all have biases. Some because a 
representative model of some kind will almost surely be used to decide the 
categories of people to be included on the review team and people 1 s experiences 
tend to influence their perceptions (a strength as well as a weakness). Third, 
it is easy to wish to be associated with success and as easy to wish to be 
known as an advocate for gooa causes (e.g., educational equity). This can lead 
to "band-wagon 11 support or to hyper-critical reporting. 

Fourth, the review team was dependent on personal reports to provide the 
evidentiary material for its deliberations. For although written evidence and 
field observations were also employed, they were not designed for the audit 
itself (as the interviews were) and, while indispensible, provided an 
incomplete sense of the district's educational program. As a consequence of 
this dependency, one of the problems confronting the review team was 
establishing the validity of the perspectives offered by those being 
interviewed. For example, district employees, when confronted by an outsider, 
often tend to "circle the wagons, " responding in ways that protect the 
district, and even to react defensively rather than openly. Others may view 
the audit as an opportunity to serve a personal agenda through selective 
"testimony. 11 The review team handled this problem by attempting to 
"triangulate" by probing a variety of sources about each concern. In addition, 
some people were interviewed more than once, to explore new issues or to 
revisit those about which the review team had new evidence or further 
questions. Finally, a wide variety of types of evidence (e.g., statistical 
analyses, research and evaluations, in addition to the self-report information) 
were obtained and examined. Limitations on the review team's ability to use 
these measures included the amount of time available on site and the fact that 
knowledge of the particular details about any district program accumulates with 
experience in the district, which uses that time. 

Finally, it is easy to try to outguess the media or others who it is believed 
will "use" the report in some fashion. While it is not certain that this 
review tsam avoided all these problems, the members of the review team were 
well aware of them and steps were taken to address them. The superintendent 
also set a tone of intellectual honesty that permitted genuine and warranted 
criticism to be expressed. In addition, jach person on the review team 
realized that mere expressions of admiration were not, in fact, useful to the 
district. Each also felt a strong obligation to represent the needs of 
students and to accept the superintendent and board presidents 1 charges to 
provide an objective appraisal of the program. Finally, members of the team 
tended to check and balance each other: those who might tend to be overly 
laudatory of an activity were often balanced by someone critical of it and 
vice-versa. 

In the case of the final report of the review team (Hammond, McKenzie, Myers, 
Walters, and Ward, 1989). the concern about what the media (and others) led not 
to the potential self-entrapment of second-guessing, but to the inclusion of 
the following statement: 
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In wnat follows, there will be some questions raised, some criticism 
offered, and some doubts expressed. It should be explicitly 
understood by the readers of this document that all of these not 
withstanding, the Pittsburgh Public School District is, without 
question, one of the finest urban districts in the country. We do 
not know of any other that has done so much in the past eight years. 
The accomplishments of the district are many and are remarkable. The 
people of the city of Pittsburgh are receiving excellent service from 
their schools and tnose wno lead them. They are fortunate, not 
simply because these educators and board members are doing a superior 
job, but because they continually seek to do their job even better. 

Anyone reading this report who does not include the sentiments 
expressed in the preceding paragraphs will have misinterpreted this 
report and the views of the External Review Team (Hammond, et al, 
1989. p. i.). 

Examined long-term change efforts - The third characteristic of this 
educational program audit was that it attempted to review activities that 
developed and evolved over an eight-year period (from the beginning of the 
Wallace super intendency to date). This suited the comprehensive nature of the 
audit, and gave a depth to the study that was desirable. The combination of 
comprehensiveness and longitudinal perspective meant that the team reviewed the 
collection of programs that had accumulated in the district with a particular 
concern for connections and contradictions between and among them. The team 
members realized that programs evolve in response to outside pressures and 
trends. Once in place, programs may then continue on their own despite 
changing needs in the district and its community. The educational program 
audit brings a fresh perspective with its review and allows questions about the 
continued relevance of programs to be raised. Finally, the longitudinal nature 
of the audit means that trends may be observed and anticipated. In this case, 
the review team brought to the district's attention the fact that an announced 
priority, discipline, was not being treated as a priority. Further, the team 
pointed to the problematic interaction between the neglect of this area and the 
next steps that were called for as the district continued to address the 
priority of student achievement which had been its prime focus. 

The longitudinal nature of the audit also-added to the problems of perspective 
and veracity, because some actions were not recent, and because some of the 
actors (only a few, thankfully) who had played major roles were no longer with 
the district at the time of the study. 

The longitudinal emphasis adds a helpful perspective to the study, but for a 
variety of reasons (e.g., the decision-oriented nature of the audit) there is a 
tendency to focus on the present. This is benefi-.ial to the extent that it 
forces attention on constructive action, but it may cause or exacerbate a 
tendency to dwell on problems and to overlook accomplishments. In this audit, 
for example, the accomplishments of the basic skills program could have been 
overlooked. The questions about higher cognition, and about social, cultural, 
and emotional factors are second-generation questions. That is, while they are 
real and immediate issues for Pittsburgh, they derive from the base already 
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established and ask> in effect, how will the Pittsburgh schools move on from 
here? This perspective is more readily achieved in a longitudinal and 
synthetic view. 

Thoughts About Conducting an Educational Program Audit 

Appropriate Questions - As was noted earlier, an educational program audit, 
using the most basic definition, occurs to verify claims made by a 
practitioner. The appropriate questions for such an audit have to do with the 
accuracy of the claims in a literal sense — do the claims make sense; does a 
sampling of relevant evidence support them; were the procedures used to develop 
the claims "state-of-the-art;" is this district, in short, making a proper 
accounting to the public? Questions that elicit evidence evaluating such 
claims are clearly within the purview of a program audit. The nature of this 
audit took the review team far beyond such narrow questions. Indeed it is 
highly unlikely that the people who carried out the audit would have considered 
doing so had they not felt free to explore beyond narrow questions in the 
interest of the broader good of educational improvement for Pittsburgh. The 
members of the review team were aware that they were selected for their 
personal expertise. This not only allowed professional judgment to be applied, 
but expected, encouraged, and valued it. 

Under these circumstances, review team members explored topics with district 
staff members in a give-and-take fashion. Questions about the place of child 
development in the district's program, effective discipline programs, parent 
involvement, and ways to encourage students to go on to college were examples 
of those investigated within the general framework or mission of the audit. 
Thus, the educational program audit also undertook to explore issues of 
critical importance to the district in its pursuit of an effective definition 
and delivery of its educational programs. 

Use of the Results - As of this writing, it is not possible to document all of 
the ways in which the results of the educational program audit have been or 
will be used. Two things, however, seem clear. First, the audit succeeded in 
its essential mission to investigate the credibility of the claims of 
accomplishment by the district. Second, there are many more uses beyond that 
relatively narrow one for which it will eventually prove beneficial. 

- z — As described earlier, this educational program audit had as its primary <:oal 
the investigation of whether the apparent successes of the school district 
could be validated rationally and ir the judgment of an external review team of 
respected professionals. The district routinely investigated student 
achievement and had for several years reported gratifying increases in test 
scores. Moreover, many of the district's programs were often cited as 
exemplary and of national stature. Further, the system had taken care to audit 
the quality of its test results, repeatedly finding them to be a veritical 
reflection of student achievement throughout the district. In addition to all 
this, however, a final, and in seme ways more convincing, element of its public 
accounting required an investigation of the quality of its programs. This 
would reassure the school board and the public that beyond the numbers in the 
reports of test scores, there was both a vision of educational reform and 
programming of sufficient quality to make those numbers plausible. 
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The educational program audit succeeded in this regard. The school beard has 
taken to heart the findings of the audit. A news release has been made, 
summarizing the results of the audit. Articles have been published in the 
local newspapers and major television and radio stations have carried the 
story, even interviewing review team members on the air. The board and the 
administration have made references to it whenever questions of programming 
quality have come up. 

Beyond this rather narrow accountability function, however, lies a more 
profound use of the audit results, namely ins instrumental use as a motivator 
for change and continued reform within tne district. Pittsburgh, like many 
other districts throughout the country, lias been jotive in its pursuit of 
educational reform over the past decade. In a number of areas, it has realized 
significant successes. However, it is possible, even likely, that after a 
certain period of attending to this consuming work the energies that fuel the 
reform movement might begin to wane (a point made in the final report itself). 

Flagging energies do not diminish the challenge. Urban education, even after a 
considerable measure of success, remains fraught with problems that require a 
re-energizing of those whose responsibility it is. While no conclusive 
observations may be offered at this time, there are signs that the educational 
program audit may assist here. 

The school board has analyzed issues using the terms and perspectives of the 
audit. For example, through the audit, the board realized t' at it had 
neglected the priority it had set for improving student discipline. By the 
time the audit was concluded, and as a result of it, the district was already 
taking steps to reorganize and stabilize the administrative unit responsible 
for student discipline. 

Of even greater instrumental impact is the role the audit is playing within the 
administration's own discussions about policy and programs. Each of the 
several administrative divisions whose work was reviewed in the audit received 
copies of the report. Division personnel then devoted time to its review and 
discussion. They are now engaged in identifying those ways in which their own 
efforts can be made to respond to the constructive criticisms and rich 
suggestions of the report. 

General Promises and Pitfalls - One key to the success, or at least the 
harmony, of this educational program audit was that the review team consisted 
of individual? who got along well with each other. This was a blessing for 
which complete responsibility cannot be claimed or assigned. While selection 
of the actual review team members is at least partly controllable (there is a 
political element in both the categories of persons and the actual individuals 
chosen), team "chemistry," is much less so — indeed it is hardly predictable at 
all. 

The district brought in outside experts and let them go where they wished, see 
what they wanted to and write whan they would, all with a minimum of influence 
exercised. Draft reports were reviewed by district staff members. Three sorts 
of comments were offered: (1) facts that were in error were pointed out and 
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corrections offered, (2) occasional wording that was problematic because of 
district history was pointea out along lath a request for a cnange, and (3) 
questions regarding the intent were explored. In each case f care was take to 
respect the intentions of the review team. 

The result was that the aistrict received a validation of its overall program. 
Further, many of the specific details were also validated, although some others 
were called into question. As important, some of the problems that were sensed 
by the superintendent, such as a concern about overburdening staff, were 
examined in a way that neither district staff nor any single evaluation could 
have. The audit pointed out additirnal resources that the district might call 
upon. These included (1) suggestions about making use of the expertise of 
particular principals who were heading schools in wnien black students achieved 
to assist the leadership of schools with large racial achievement gaps, (2) 
alerting the superintendent to a person to fill a significant vacancy on the 
district staff, and (3) pointing out a major report that had been provided to 
the district, but had not been fully used, as a resource critical to the next 
steps the district needed to take. 

Another virtue of the audit was that the review team members enlarged the focus 
of the conversations going on within the district. They did this partly 
because no member felt restricted to her/his particular area of recognized 
expertise. In addition, each listened to the questions that the others asked 
and incorporated their concerns and insights within his/he: 11 cwn and deepened 
them. The review team presented the opportunity for a thou^>itful exchange. 

Finally, each review team member felt an obligation to make a contribution to 
those who were taking the time to work with them. An example of this occurred 
at one high school. The school had a new principal. When the review team 
members began to talk with a group of students who had been gathered in the 
principal's office, she asked if it would be appropriate to stay. The review 
team was pleased to have her stay. It was obvious that she was most impressed 
by the presence of former Washington, DC Superintendent Floretta Dukes 
McKenzie. During the course of the discussion, McKen2ie provided the sort of 
modeling that all were seeking. One of the students was a young lady with 
obvious ability. She was, however, shy and very nervous as she contemplated 
her future. No one in her family had ever gone to college, but clearly she 
could and should go. With no one to show her the way and a family with no 
extra money for the expense of college applications, she was lonely and 
uncertain. Finally, McKen2ie took out her checkbook and wrote out a check for 
the first application fee and gave it to the principal, in trust, for the young 
lady. That is the best an audit can be. By itself, it does not justify the 
expense and arrangements needed, but it represents the heights that can be 
reached. 
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Appendix 



Findings 

The basic finding of the review team was that the people of Pittsburgh and, 
more importantly, their children are well served by their public schools. The 
district's leadership is strong and caring. Its priorities and programs are 
good ones. The teachers, principals, and other staff members we met are 
outstanding professionals. The district is headed in the right direction, 
using systematically developed plans that assure that students will be 
increasingly well served in the days and years to come. The Team reached 
general conclusions in each of the four areas Identified in the Mission 
Statement and more specific M findings and recommendations 11 in the four areas of 
focus negotiated with the board and superintendent, 

K_ The Priorities: The External Review Team believes that the board's 
priorities are appropriate and well-conceived. In 1986, the board wisely 
reviewed and revised its 1981 priorities to reflect changing times, emerging 
needs, and progress made by the schools. These modifications did not signal a 
major change in direction, but instead, refocused the district's resources 
within previously established priorities such as Achievement and Student 
Discipline. 

I I , The Programmatic Responses of the Administration: In response to the 
board's priorities, in both 1981 and 1986, the administration engaged district 
personnel and some community representatives in a systematic planning process. 
Representative committees carefully studied the major components of each 
priority area and prepared comprehensive, research-based plans for 
implementation of programs to carry out the priorities. 

III , The Implementation and Evolution of Programs: Program plans have been 
implemented systematically and in a manner consistent with the board's 
priorities. Allowance was made for review, revision, and adaptation to 
individual circumstances that arose in varying contexts. 

IV, The Status of the Overall Educational Program Compared to Other Public 
School Districts: The Pittsburgh Public School District has, without question, 
established itself as one of the outstanding urban districts in the country. 

• r — We do not know of any other district that^has accomplished so much in the past 
eight years. The district has avoided the temptation of the "quick fix" and 
has resisted finger pointing. It has unified its leadership and its staff and 
become fully engaged in meeting its educational responsibilities. The people 
of the city of Pittsburgh are receiving excellent service from their schools 
and those who lead them. They are fortunate, not simply because the community 
educators are doing an exceJient job, but because they continually seek to do 
their Job even better. 
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The major findings and recommendations of the External Review Team are: 
Student Achievement 

1. Since 1980, the standardized test scores reported by the district in 
reading, mathematics, and language have shown major gains in student 
achievement in the Pittsburgh Public Schools. Because the district does 
not administer standardized tests to its students in science, changes are 
not known in that area. 

Recommendat ions : (a) Maintain and strengthen the programs that have 
produced the gains, (b) Conduct an evaluation to determine the district's 
success in "increasing the quality and pacing of. . . science courses." 

2. The district's impressive gains on achievement tests may be reaching a 
plateau because: (a) the district's main instructional program is so 
focused on basic skills that higher order thinking is relatively 
neglected, (b) the energy that has motivated program implementation to 
date needs renewal, arid (c) the achievement tests used to assess the gains 
are limited measures of student achievement. 

Recommendation: To reacn the next level, the district will have to design 
a more comprehensive program, one that includes higher order cognition, 
affective, social, and cultural components that engage and support the 
multiracial, multicultural urban community it serves. The elements of 
such an approach exist in plans written in 1987 by the district's 
committees on the racial achievement gap and low achieving students, and 
they should be used. Program implementation should occur after a time of 
review, reflection, and communication across the district (i.e., between 
and an.ong various school and central office units). This will enable new 
clarity and unity about goals that will restore che staff's energy. The 
new goals must focus on individual school differences as a source of 
strength as well as of need. Means of assessment must be devised that 
respond to the complexity of this new approach. 

3. The district has made substantial gains in assisting low achieving 
students, as evidenced by the low proportions of students at all grade 
levels scoring below the 25th percentile. 

Recommendation: The district should continue its attention to programs 
serving students scoring below the 25th percentile. 

4. A racial achievement gap divides black and white students 1 teat res. 
While progress was made in reducing that gap from 1979 to 1984, there has 
been little change since then in reading and language. In mathematics, 
the gap has widened by five points since 1984. The pilot programs the 
district has in place are supplementary in nature and, therefore, 
insufficient to close the gap even if fully implemented throughout the 
district. 

Recommendation : Closing the racial achievement gap must be made a 
priority by the board, the superintendent, and the district staff. To 

2 
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close it, a comprehensive program like that described in item 2 above will 
be required. 

5. A comprehensive staff development program founded on a belief that 

constructive support is a productive means of improving instruction and 
leadership has been a critical and fundamental ingredient of the 
district 1 s success ♦ 

Recommendation; The board must continue to fund and encourage staff 
development programs that enhance the skills of district staff memDers as 
they undertake new and more complex efforts to improve instruction in 
Pittsburgh. PRISM should be strengthened by focusing on subject matter 
content more than on general instructional methods and by emphasizing the 
knowledge, understanding, and behaviors necessary to close the racial 
achievement gap and to continue to improve instruction for low achieving 
students. 

Student Discipline 

1. The district has established and disseminated a consistent district-wide 
discipline code and set of procedures. 

Recommendation: Review the use of these procedures to assure that they 
are equitably applied in all district schools. 

2. The district is piloting a set of programs designed to prevent serious 
discipline problems, dropouts, suspensions, and expulsions. 

Recommendation: These programs must be evaluated to assess the extent to 
which they are achieving this purpose. 

3. The high levels of suspensions, and emphasis on rote learning tasks in 
detention and in-school suspension programs are indicative of an 
uncharacteristically narrow and punitive approach to discipline in the 
district. 

Recommendation : The district must develop a comprehensive discipline 
program that, like the achievement program proposed above, includes 
affective, social, and cultural components that engage and support its 
multiracial, multicultural urban community. 

4. The district's commitment to addressing discipline issues is unevenly 
implemented. The organization and Wdership of the Pupil Services 
department has changed several times in the past eight years and these 
changes have weakened its potential effectiveness. 

Recommendations : 

a. Appoint a department head who is recognized as a leader and will 
serve long-term with a mandate to direct the districts total 
discipline effort. 
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b. Establish an electronic ' \ base for collecting and reporting all 
discipline-related information disaggregated by gender, ethnicity, 
offense, and location. 

c. Report district data regularly to the soard and superintendent, 
including analyses and recommendations about emerging problems as 
well as progress. 

d. Replace the overuse of out-of-school suspensions and transferring 
students to other schools with alternatives that respond to high 
school students who are finding school a difficult place to be. 

Staff Evaluation 

1. PRISM was designed as a staff evaluation program for both principals and 
teachers. Its in-service included research-based guidelines for promoting 
effective instruction and school leadership. This simultaneous attention 
to evaluation and to the elements of school and instructional 
effectiveness, involving both principals and teachers, established the 
comprehensive approach that led to improved instruction district-wide. 

Recommendation: A constructive approach to staff evaluation, based on a 
clear model, must be maintained and extended in the future. 

2. Instructional Team Leaders (ITLs) and Instructional Cabinets are promising 
approaches to strengthening school site and shared decision making. 

Recommendation : To avoid these innovations becoming routine rep3 icas of 
old structures (such as department heads), their responsibilities must be 
challenging and clear. For example, making Instructional Cabinets 
accountable for closing the racial achievement gap or reducing the numbers 
of suspensions at their schools would give them a specific role central to 
the priorities of the district. For the ITLs to become effective leaders 
in their schools, staff development will be necessary. 

3. A positive approach to staff evaluation has been evident in the unique 
relationship between and among the board, superintendent, and teachers 1 
organization (Pittsburgh Federation of Teachers). 

Recommendation: Continue the hard work that has resulted in this 
productive relationship, and extend the results still further. 

4. A negative tone appears to characterize the board's relationship with the 
district's principals and administrators. There is an air of mistrust and 
uncertainty that is inconsistent with the constructive programs and 
support that have been the foundation of the district's accomplishments. 

Recommendation: The board and administration need to work with the 
administrative leadership in a manner like that used with the Pittsburgh 
Federation of Teachers. Using a positive approach similar to that which 
characterizes the PRISM program wouln allow all four groups to work 
together to move the district to yet a higher level than has been 
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achieved. They must share and develop leadership skills at the district, 
school, and classroom levels that unite the system. 



Attracting and Holding Students 

1. The district has made excellent strides in attracting students to its 
schools. 

Recommendation : This accomplishment is a by-proauct of the districts 
achievement program. As that program continues to improve, more students 
will be attracted to the Pittsburgn Public Schools. 

2 The district's efforts to hold scuaents are mixed. Enrollment decline has 
ended, but dropouts and suspensions remain problems. 

Recommendation : The district's accomplishments in this area will be 
determined by tne successes of the improved achievement an»J discipline 
programs recommended above. 
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